Internal resource provisioning in database systems

ABSTRACT

Resource provisioning systems and methods are described. In an embodiment, a system includes a plurality of shared storage devices collectively storing database data, an execution platform, and a compute service manager. The compute service manager is configured to determine a task to be executed in response to a trigger event and determine a query plan for executing the task, wherein the query plan comprises a plurality of discrete subtasks. The compute service manager is further configured to assign the plurality of discrete subtasks to one or more nodes of a plurality of nodes of the execution platform, determine whether execution of the task is complete, and in response to determining the execution of the task is complete, store a record in the plurality of shared storage devices indicating the task was completed.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. patent applicationSer. No. 16/778,954 filed Jan. 31, 2020, which is a continuation of U.S.patent application Ser. No. 16/380,848, filed on Apr. 10, 2019, thecontents of which are incorporated herein by reference in theirentireties.

TECHNICAL FIELD

The present disclosure relates to systems, methods, and devices fordatabases and more particularly relates to resource management relatedto data processing and data storage.

BACKGROUND

Databases are an organized collection of data that enable data to beeasily accessed, manipulated, and updated. Databases serve as a methodof storing, managing, and retrieving information in an efficient manner.Traditional database management requires companies to provisioninfrastructure and resources to manage the database in a data center.Management of a traditional database can be very costly and requiresoversight by multiple persons having a wide range of technical skillsets.

Traditional relational database management systems (RDMS) requireextensive computing and storage resources and have limited scalability.Large sums of data may be stored across multiple computing devices. Aserver may manage the data such that it is accessible to customers withon-premises operations. For an entity that wishes to have an in-housedatabase server, the entity must expend significant resources on acapital investment in hardware and infrastructure for the database,along with significant physical space for storing the databaseinfrastructure. Further, the database may be highly susceptible to dataloss during a power outage or other disaster situations. Suchtraditional database systems have significant drawbacks that may bealleviated by a cloud-based database system.

A cloud database system may be deployed and delivered through a cloudplatform that allows organizations and end users to store, manage, andretrieve data from the cloud. Some cloud database systems include atraditional database architecture that is implemented through theinstallation of database software on top of a computing cloud. Thedatabase may be accessed through a Web browser or an applicationprogramming interface (API) for application and service integration.Some cloud database systems are operated by a vendor that directlymanages backend processes of database installation, deployment, andresource assignment tasks on behalf of a client. The client may havemultiple end users that access the database by way of a Web browserand/or API. Cloud databases may provide significant benefits to someclients by mitigating the risk of losing database data and allowing thedata to be accessed by multiple users across multiple geographicregions.

Databases are widely used for data storage and access in computingapplications. A goal of database storage is to provide enormous sums ofinformation in an organized manner so that it can be accessed, managed,and updated. In a database, data may be organized into rows, columns,and tables. Different database storage systems may be used for storingdifferent types of content, such as bibliographic, full text, numeric,and/or image content. Further, in computing, different database systemsmay be classified according to the organization approach of thedatabase. There are many different types of databases, includingrelational databases, distributed databases, cloud databases,object-oriented and others.

Queries can be executed against database data to find certain datawithin the database and respond to a question about the database data. Adatabase query extracts data from the database and formats it into areadable form. For example, when a user wants data from a database, theuser may write a query in the language required by the database. Thequery may request specific information from the database. For example,if the database includes information about sales transactions made by aretail store, a query may request all transactions for a certain productduring a certain time frame. The query may request any pertinentinformation that is stored within the database. If the appropriate datacan be found to respond to the query, the database has the potential toreveal complex trends and activities. This power can only be harnessedthrough the use of a successfully executed query.

Many existing data storage and retrieval systems are available today.For example, in a shared-disk system, all data is stored on a sharedstorage device that is accessible from all of the processing nodes in adata cluster. In this type of system, all data changes are written tothe shared storage device to ensure that all processing nodes in thedata cluster access a consistent version of the data. As the number ofprocessing nodes increases in a shared-disk system, the shared storagedevice (and the communication links between the processing nodes and theshared storage device) becomes a bottleneck that slows data read anddata write operations. This bottleneck is further aggravated with theaddition of more processing nodes. Thus, existing shared-disk systemshave limited scalability due to this bottleneck problem.

Another existing data storage and retrieval system is referred to as a“shared-nothing architecture.” In this architecture, data is distributedacross multiple processing nodes such that each node stores a subset ofthe data in the entire database. When a new processing node is added orremoved, the shared-nothing architecture must rearrange data across themultiple processing nodes. This rearrangement of data can betime-consuming and disruptive to data read and write operations executedduring the data rearrangement. And, the affinity of data to a particularnode can create “hot spots” on the data cluster for popular data.Further, since each processing node performs also the storage function,this architecture requires at least one processing node to store data.Thus, the shared-nothing architecture fails to store data if allprocessing nodes are removed. Additionally, management of data in ashared-nothing architecture is complex due to the distribution of dataacross many different processing nodes.

The systems and methods described herein provide an improved approach todata storage and data retrieval that alleviates the above-identifiedlimitations of existing systems.

BRIEF DESCRIPTION OF THE DRAWINGS

Non-limiting and non-exhaustive embodiments of the present disclosureare described with reference to the following figures, wherein likereference numerals refer to like parts throughout the various figuresunless otherwise specified.

FIG. 1 is a block diagram illustrating an example process flow forscheduling tasks on a database, according to one embodiment of thedisclosure;

FIG. 2 is a block diagram illustrating a data processing platform,according to one embodiment of the disclosure;

FIG. 3 is a block diagram illustrating a compute service manager,according to one embodiment of the disclosure;

FIG. 4 is a block diagram illustrating an execution platform accordingto one embodiment of the disclosure;

FIG. 5 is a block diagram illustrating an example operating environmentaccording to one embodiment of the disclosure;

FIG. 6 is a block diagram illustrating an example lifecycle of a taskaccording to one embodiment of the disclosure;

FIG. 7 is a block diagram illustrating an example process flow forscheduling and executing tasks on a database, according to oneembodiment of the disclosure;

FIG. 8 is a schematic flow chart diagram illustrating a method forscheduling tasks to be executed on a database, according to oneembodiment of the disclosure;

FIG. 9 is a schematic flow chart diagram illustrating a method forscheduling tasks to be executed on a database, according to oneembodiment of the disclosure; and

FIG. 10 is a block diagram illustrating an example computing device,according to one embodiment of the disclosure.

DETAILED DESCRIPTION

The systems, methods, and devices described herein provide a new meansfor scheduling and executing tasks on shared storage and executionplatforms. The systems, methods, and devices described herein may beimplemented on multiple tenant cloud-based database platforms. In someinstances, it may be desirable to execute “internal” tasks on thedatabase to improve database operations or optimize database storage.Such internal tasks may be triggered by a time schedule and/or someprogrammatic logic that is triggered by an update or other change to thedatabase. Such internal tasks may not be received directly from a clientbut may instead be configured to providing behind-the-scenesoptimizations to the database. Such optimizations may include, forexample, reclustering a table, refreshing a materialized view,propagating an update to one or more replications of data, updating achange tracking summary, and so forth. Such internal tasks may begenerated and executed in a platform that is effectively infinitelyscalable to ensure acceptable task latency and task throughput.

Such systems, methods, and devices provide a low effort and low overheadmeans to build new services in a database system. Such services mayinclude, for example, an automated data clustering service, an automatedmaterialized view refresh service, a file compaction service, a storageprocedure service, a file upgrade service, and so forth. Further, thedisclosures herein provide means to execute internal jobs that improvequery performance of the database and/or improve data organization inthe database. Further, the disclosures herein provide means to runqueries on behalf of a client account and to view, manage, and audit oneor more discrete tasks associated with a job. Additionally, thedisclosures herein provide for built-in automatic scaling of processingresources for asynchronously executing one or more discrete tasks.

In an embodiment, a compute service manager schedules and manages theexecution of a job by separating the job into one or more discretetasks. The compute service manager may convert scheduled work (i.e. a“job”) into a plurality of discrete tasks and manage the asynchronousexecution of those discrete tasks by an execution platform. The computeservice manager may schedule and manage the execution of tasks for avariety of implementations and may be particularly suited for schedulingthe execution of tasks for a clustering service, a materialized viewrefresh service, a file compaction service, a storage procedureexecution service, and a file upgrade service. The compute servicemanager may be particularly implemented for scheduling and managing theexecution of internal “behind the scenes” jobs that are not directlyreceived from a client account. Such internal jobs may improve thefunctionality or organization of database systems by, for example,reclustering data, incrementally refreshing a materialized view based ona source table, and so forth.

It should be appreciated that the compute service manager may manage theexecution of any number of jobs or type of jobs. In an embodiment, thecompute service manager is particularly suited to managing the executionof internal “behind the scenes” jobs that are not visible to a clientaccount. Such internal jobs include, for example, table reclustering andthe automated refresh of a materialized view.

The compute service manager may manage and schedule jobs within varioususer-specified restraints. For example, a client account may specifyretry constraints that indicate a number of times a job may bere-executed and when the job should be re-executed. A client account mayuniquely tag a job and indicate that certain jobs should be executed oncertain database tables. In an embodiment, the compute service managerpermits a client account to search scheduled work by any suitableparameter, including for example, account identification, individualwork item identification name, timestamp, or type of work. A clientaccount may indicate that a single work item should be converted intoone or more jobs and/or that a single job should be converted into oneor more discrete tasks.

In an embodiment, the compute service manager programmatically spawnsjobs to execute arbitrary structured query language (SQL) commands. Insuch an embodiment, the compute service manager may look at a set ofparameters to determine if an “internal” job or a “customer-facing” jobneeds to be performed.

The systems and methods described herein provide a new platform forstoring and retrieving data without the problems faced by existingsystems. For example, this new platform supports the addition of newnodes without the need for rearranging data files as required by theshared-nothing architecture. Additionally, nodes can be added to theplatform without creating bottlenecks that are common in the shared-disksystem. This new platform is always available for data read and datawrite operations, even when some of the nodes are offline formaintenance or have suffered a failure. The described platform separatesthe data storage resources from the computing resources so that data canbe stored without requiring the use of dedicated computing resources.This is an improvement over the shared-nothing architecture, which failsto store data if all computing resources are removed. Therefore, the newplatform continues to store data even though the computing resources areno longer available or are performing other tasks.

In the following description, reference is made to the accompanyingdrawings that form a part thereof, and in which is shown by way ofillustration specific exemplary embodiments in which the disclosure maybe practiced. These embodiments are described in sufficient detail toenable those skilled in the art to practice the concepts disclosedherein, and it is to be understood that modifications to the variousdisclosed embodiments may be made, and other embodiments may beutilized, without departing from the scope of the present disclosure.The following detailed description is, therefore, not to be taken in alimiting sense.

Reference throughout this specification to “one embodiment,” “anembodiment,” “one example” or “an example” means that a particularfeature, structure or characteristic described in connection with theembodiment or example is included in at least one embodiment of thepresent disclosure. Thus, appearances of the phrases “in oneembodiment,” “in an embodiment,” “one example” or “an example” invarious places throughout this specification are not necessarily allreferring to the same embodiment or example. In addition, it should beappreciated that the figures provided herewith are for explanationpurposes to persons ordinarily skilled in the art and that the drawingsare not necessarily drawn to scale.

Embodiments in accordance with the present disclosure may be embodied asan apparatus, method or computer program product. Accordingly, thepresent disclosure may take the form of an entirely hardware-comprisedembodiment, an entirely software-comprised embodiment (includingfirmware, resident software, micro-code, etc.) or an embodimentcombining software and hardware aspects that may all generally bereferred to herein as a “circuit,” “module” or “system.” Furthermore,embodiments of the present disclosure may take the form of a computerprogram product embodied in any tangible medium of expression havingcomputer-usable program code embodied in the medium.

Any combination of one or more computer-usable or computer-readablemedia may be utilized. For example, a computer-readable medium mayinclude one or more of a portable computer diskette, a hard disk, arandom-access memory (RAM) device, a read-only memory (ROM) device, anerasable programmable read-only memory (EPROM or Flash memory) device, aportable compact disc read-only memory (CDROM), an optical storagedevice, and a magnetic storage device. Computer program code forcarrying out operations of the present disclosure may be written in anycombination of one or more programming languages. Such code may becompiled from source code to computer-readable assembly language ormachine code suitable for the device or computer on which the code willbe executed.

Embodiments may also be implemented in cloud computing environments. Inthis description and the following claims, “cloud computing” may bedefined as a model for enabling ubiquitous, convenient, on-demandnetwork access to a shared pool of configurable computing resources(e.g., networks, servers, storage, applications, and services) that canbe rapidly provisioned via virtualization and released with minimalmanagement effort or service provider interaction and then scaledaccordingly. A cloud model can be composed of various characteristics(e.g., on-demand self-service, broad network access, resource pooling,rapid elasticity, and measured service), service models (e.g., Softwareas a Service (“SaaS”), Platform as a Service (“PaaS”), andInfrastructure as a Service (“IaaS”)), and deployment models (e.g.,private cloud, community cloud, public cloud, and hybrid cloud).

The flow diagrams and block diagrams in the attached figures illustratethe architecture, functionality, and operation of possibleimplementations of systems, methods, and computer program productsaccording to various embodiments of the present disclosure. In thisregard, each block in the flow diagrams or block diagrams may representa module, segment, or portion of code, which comprises one or moreexecutable instructions for implementing the specified logicalfunction(s). It will also be noted that each block of the block diagramsand/or flow diagrams, and combinations of blocks in the block diagramsand/or flow diagrams, may be implemented by special purposehardware-based systems that perform the specified functions or acts, orcombinations of special purpose hardware and computer instructions.These computer program instructions may also be stored in acomputer-readable medium that can direct a computer or otherprogrammable data processing apparatus to function in a particularmanner, such that the instructions stored in the computer-readablemedium produce an article of manufacture including instruction meanswhich implement the function/act specified in the flow diagram and/orblock diagram block or blocks.

The systems and methods described herein provide a flexible and scalabledata warehouse using a new data processing platform. In someembodiments, the described systems and methods leverage a cloudinfrastructure that supports cloud-based storage resources, computingresources, and the like. Example cloud-based storage resources offersignificant storage capacity available on-demand at a low cost. Further,these cloud-based storage resources may be fault-tolerant and highlyscalable, which can be costly to achieve in private data storagesystems. Example cloud-based computing resources are available on-demandand may be priced based on actual usage levels of the resources.Typically, the cloud infrastructure is dynamically deployed,reconfigured, and decommissioned in a rapid manner.

In the described systems and methods, a data storage system may utilizean SQL (Structured Query Language)-based relational database. However,these systems and methods are applicable to any type of database, andany type of data storage and retrieval platform, using any data storagearchitecture and using any language to store and retrieve data withinthe data storage and retrieval platform. The systems and methodsdescribed herein further provide a multi-tenant system that supportsisolation of computing resources and data between differentcustomers/clients and between different users within the samecustomer/client.

Turning to the figures, FIG. 1 is a block diagram of an exampleembodiment of a process flow 100 for managing and executing jobs on adatabase system. In an embodiment, the process flow 100 is carried outby a compute service manager 102 that is configured to manage andexecute jobs on a new data processing platform (see, e.g. FIGS. 2-5).The compute service manager 102 receives a job 104 that may be dividedinto one or more discrete tasks, e.g. task 0, task 1, task 2, task 3,and so forth through task (n). The compute service manger 102 receivesthe job at 106 and determines tasks at 108 that may be carried out toexecute the job 104. The compute service manager 102 is configured todetermine the one or more tasks, such as task 0, task 1, task 2, task 3,and so forth, based on applicable rules and/or parameters. The computeservice manager assigns tasks at 110. In an implementation, the computeservice manager 102 may assume a client role at 112 to act on clientdata. The job 104 is carried out by a query manager 114 of the computeservice manager 102. The query manager 114 may have multiple threads,including for example query manager threads 114 a, 114 b, 114 c, and soforth. The compute service manager 102 may provide the job 104,including the multiple discrete tasks, to the execution platform 116 forthe job to be executed. The compute service manager 102 may assign eachof the multiple discrete tasks to various execution nodes of theexecution platform 116.

The compute service manager 102 determines one or more discrete tasks tobe executed to carry out a job 104. A task is a portion of work that maybe scheduled for execution by the execution platform 116. The job 104may include a state that can be serialized for storage, a means todeserialize that state, and a set of methods that operate based on thatstate to produce the one or more discrete tasks and make decisionsconcerning how to deal with, for example, errors, failures, statetransitions, and so forth. The state may reside in persistent storageand may be updated to reflect a job that is executed by the computeservice manager 102.

In an embodiment, the compute service manager 102 schedules and managesthe execution of queries on behalf of a client account. The computeservice manager 102 may schedule any arbitrary SQL query. The computeservice manager 102 may assume a role to schedule the job 104 as if itis the client account rather than as an internal account or otherspecial account. The compute service manager 102 may embody the role of,for example, an account administrator or a role having the smallestscope necessary to complete the intended job 104. In an embodiment, thecompute service manager 102 embodies the role that owns the object thatis the target of the job 104, e.g. for a cluster, the table beingclustered. In an embodiment, the compute service manager 102 receivesthe job 104 and the job 104 specifies a domain or identification of theobject that will be operated on. From that domain or identification, thecompute service manager 102 resolves the object and assumes theappropriate role identification. In an embodiment, the compute servicemanager 102 assumes a role of a special “compute service user” that isnot visible to or usable by a client account.

The compute service manager 102 determines tasks at 108 and assignstasks at 110. The compute service manager 102 generates one or morediscrete units of work that may be referred to as a task. The taskincludes, for example, a task type, a task identification, an accountidentification, a payload which may be converted to one or more discretetasks, and a set of options that control how the job 104 behaves (e.g.indicates a number of retries). The task includes a “state” thatidentifies where in the task lifecycle a given task is. The persistentstate for a task may be split between values that are hard coded intopersistent data storage and a task implementation-specific object thatis deserialized when operating on the task object in memory. This maypermit the task persistent data storage to be relatively simple whenpermitting individual implementers to store arbitrary information abouttheir associated tasks. The task namespace includes an index class thatgroups information for concrete implementations of a scheduled task,including a class object for the concrete implementation, an objectdomain for the task, and any other information that is pertinent to thetask type.

The compute service manager 102 may generate and assign a taskcontinuation and/or a child task. In certain implementations, a task mayrequire multiple iterations before converging to a completed state. Insuch an implementation, a task may only be a portion of a larger pieceof work that requires running many such tasks serially to be completed.This may be accommodated by generating successor tasks that areinitiated when a task is successfully completed. The successor task isatomically scheduled during state transition from “executing” (see 608in FIG. 6) to “completed success” (see 610 in FIG. 6) and is marked withthe identification of the parent task such that a lineage for the taskmay be reviewed. A callback for a successor task may be run with thesame account, role, user, and/or session of the parent task that it isbeing called for.

In an embodiment, the job 104 is fail-safe and has a life cycle thatbegins when the one or more discrete tasks are scheduled to be executedand/or assigned to the execution platform 116. In an embodiment, the oneor more discrete tasks may be scheduled via an application programinterface (API).

In an embodiment, the compute service manager 102 receives a job at 108by way of an internal mechanism, and the job 104 is not receiveddirectly from a client account. In an alternative embodiment, the job104 is received directly from a client account. Where the job 104 isdetermined and received by way of an internal mechanism, the job 104 mayinclude a “behind the scenes” operation that improves the management ororganization of database data. Such internal jobs include, for example,clustering or reclustering database data, refreshing a materialized viewbased on an updated source table, compacting one or more database tablesor micro-partitions, executing a storage procedure service, andupgrading files or micro-partitions in database table. The computeservice manager 102 may receive such a job at 106 based on a triggerevent. In an example implementation, where the compute service manager102 receives a job to refresh a materialized view, the trigger event maybe the updating or refreshing of the source table for the materializedview. In an additional example implementation, where the compute servicemanager 102 receives a job to recluster database data, the trigger eventmay be that a table has fallen below a predefined clustering thresholdor that new data is ingested into a database table. In animplementation, the compute service manager schedules and managesinternal jobs that improve database operations, database organization,and database query performance, and does not schedule or manage theexecution of a query (such as a SQL statement) that is received from auser or client account.

The compute service manager 102 is configured to determine one or morediscrete tasks at 108 that must be performed to fully execute the job104. In an embodiment, the one or more discrete tasks do not have anyordering constraints and may be executed in parallel or in any otherorder. In another embodiment, the compute service manager 102 assignsordering constraints to any number of the one or more discrete tasks,where applicable. Depending on the constraints of the job 104, thecompute service manager 102 may determine that one or more of multiplediscrete tasks must be serialized and executed in a particular order.

In an embodiment, the compute service manager 102 generates a reportindicating when a job 104 is scheduled to be executed and how muchcomputing resources are estimated to be tied up executing the job 104.The compute service manager 102 may generate a statement for each taskthat exposes the job 104 to an applicable client account by way of afilter. The compute service manager 102 may alert a client account whena job 104 is being executed particularly where the job 104 uses acustomer-managed key.

FIG. 2 is a block diagram depicting an example embodiment of a dataprocessing platform 200. As shown in FIG. 2, a compute service manager102 is in communication with a queue 204, a client account 208, metadata206, and an execution platform 116. In an embodiment, the computeservice manager 102 does not receive any direct communications from aclient account 208 and only receives communications concerning jobs fromthe queue 204. In particular implementations, the compute servicemanager 102 can support any number of client accounts 208 such as endusers providing data storage and retrieval requests, systemadministrators managing the systems and methods described herein, andother components/devices that interact with compute service manager 102.As used herein, compute service manager 102 may also be referred to as a“global services system” that performs various functions as discussedherein.

The compute service manager 102 is in communication with a queue 204.The queue 204 may provide a job to the compute service manager 102 inresponse to a trigger event. One or more jobs may be stored in the queue204 in an order of receipt and/or an order of priority, and each ofthose one or more jobs may be communicated to the compute servicemanager 102 to be scheduled and executed. The queue 204 may determine ajob to be performed based on a trigger event such as the ingestion ofdata, deleting one or more rows in a table, updating one or more rows ina table, a materialized view becoming stale with respect to its sourcetable, a table reaching a predefined clustering threshold indicating thetable should be reclustered, and so forth. The queue 204 may determineinternal jobs that should be performed to improve the performance of thedatabase and/or to improve the organization of database data. In anembodiment, the queue 204 does not store queries to be executed for aclient account but instead only includes stores database jobs thatimprove database performance.

The compute service manager 102 is also coupled to metadata 206, whichis associated with the entirety of data stored throughout dataprocessing platform 200. In some embodiments, metadata 206 includes asummary of data stored in remote data storage systems as well as dataavailable from a local cache. Additionally, metadata 206 may includeinformation regarding how data is organized in the remote data storagesystems and the local caches. Metadata 206 allows systems and servicesto determine whether a piece of data needs to be accessed withoutloading or accessing the actual data from a storage device.

In an embodiment, the compute service manager 102 and/or the queue 204may determine that a job should be performed based on the metadata 206.In such an embodiment, the compute service manager 102 and/or the queue204 may scan the metadata 206 and determine that a job should beperformed to improve data organization or database performance. Forexample, the compute service manager 102 and/or the queue 204 maydetermine that a new version of a source table for a materialized viewhas been generated and the materialized view has not been refreshed toreflect the new version of the source table. The metadata 206 mayinclude a transactional change tracking stream indicating when the newversion of the source table was generated and when the materialized viewwas last refreshed. Based on that metadata 206 transaction stream, thecompute service manager 102 and/or the queue 204 may determine that ajob should be performed. In an embodiment, the compute service manager102 determines that a job should be performed based on a trigger eventand stores the job in the queue 204 until the compute service manager102 is ready to schedule and manage the execution of the job.

The compute service manager 102 may receive rules or parameters from theclient account 208 and such rules or parameters may guide the computeservice manager 102 in scheduling and managing internal jobs. The clientaccount 208 may indicate that internal jobs should only be executed atcertain times or should only utilize a set maximum amount of processingresources. The client account 208 may further indicate one or moretrigger events that should prompt the compute service manager 102 todetermine that a job should be performed. The client account 208 mayprovide parameters concerning how many times a task may be re-executedand/or when the task should be re-executed.

The compute service manager 102 is further coupled to an executionplatform 116, which provides multiple computing resources that executevarious data storage and data retrieval tasks, as discussed in greaterdetail below. Execution platform 116 is coupled to multiple data storagedevices 212 a, 212 b, and 212 n that are part of a storage platform 210.Although three data storage devices 212 a, 212 b, and 212 n are shown inFIG. 2, execution platform 116 is capable of communicating with anynumber of data storage devices. In some embodiments, data storagedevices 212 a, 212 b, and 212 n are cloud-based storage devices locatedin one or more geographic locations. For example, data storage devices212 a, 212 b, and 212 n may be part of a public cloud infrastructure ora private cloud infrastructure. Data storage devices 212 a, 212 b, and212 n may be hard disk drives (HDDs), solid state drives (SSDs), storageclusters, Amazon S3™ storage systems or any other data storagetechnology. Additionally, storage platform 210 may include distributedfile systems (such as Hadoop Distributed File Systems (HDFS)), objectstorage systems, and the like.

In particular embodiments, the communication links between computeservice manager 102, the queue 204, metadata 206, the client account208, and the execution platform 116 are implemented via one or more datacommunication networks. Similarly, the communication links betweenexecution platform 116 and data storage devices 212 a-212 n in thestorage platform 210 are implemented via one or more data communicationnetworks. These data communication networks may utilize anycommunication protocol and any type of communication medium. In someembodiments, the data communication networks are a combination of two ormore data communication networks (or sub-networks) coupled to oneanother. In alternate embodiments, these communication links areimplemented using any type of communication medium and any communicationprotocol.

As shown in FIG. 2, data storage devices 212 a, 212 b, and 212 n aredecoupled from the computing resources associated with the executionplatform 116. This architecture supports dynamic changes to dataprocessing platform 200 based on the changing data storage/retrievalneeds as well as the changing needs of the users and systems accessingdata processing platform 200. The support of dynamic changes allows dataprocessing platform 200 to scale quickly in response to changing demandson the systems and components within data processing platform 200. Thedecoupling of the computing resources from the data storage devicessupports the storage of large amounts of data without requiring acorresponding large amount of computing resources. Similarly, thisdecoupling of resources supports a significant increase in the computingresources utilized at a particular time without requiring acorresponding increase in the available data storage resources.

Compute service manager 102, queue 204, metadata 206, client account208, execution platform 116, and storage platform 210 are shown in FIG.2 as individual components. However, each of compute service manager102, queue 204, metadata 206, client account 208, execution platform116, and storage platform 210 may be implemented as a distributed system(e.g., distributed across multiple systems/platforms at multiplegeographic locations). Additionally, each of compute service manager102, metadata 206, execution platform 116, and storage platform 210 canbe scaled up or down (independently of one another) depending on changesto the requests received from the queue 204 and/or client accounts 208and the changing needs of data processing platform 200. Thus, in thedescribed embodiments, data processing platform 200 is dynamic andsupports regular changes to meet the current data processing needs.

During typical operation, data processing platform 200 processesmultiple jobs received from the queue 204 or determined by the computeservice manager 102. These jobs are scheduled and managed by the computeservice manager 102 to determine when and how to execute the job. Forexample, the compute service manager 102 may divide the job intomultiple discrete tasks and may determine what data is needed to executeeach of the multiple discrete tasks. The compute service manager 102 mayassign each of the multiple discrete tasks to one or more nodes of theexecution platform 116 to process the task. The compute service manager102 may determine what data is needed to process a task and furtherdetermine which nodes within the execution platform 116 are best suitedto process the task. Some nodes may have already cached the data neededto process the task and, therefore, be a good candidate for processingthe task. Metadata 206 assists the compute service manager 102 indetermining which nodes in the execution platform 116 have alreadycached at least a portion of the data needed to process the task. One ormore nodes in the execution platform 116 process the task using datacached by the nodes and, if necessary, data retrieved from the storageplatform 210. It is desirable to retrieve as much data as possible fromcaches within the execution platform 116 because the retrieval speed istypically much faster than retrieving data from the storage platform210.

As shown in FIG. 2, the data processing platform 200 separates theexecution platform 116 from the storage platform 210. In thisarrangement, the processing resources and cache resources in theexecution platform 116 operate independently of the data storageresources 212 a-212 n in the storage platform 210. Thus, the computingresources and cache resources are not restricted to specific datastorage resources 212 a-212 n. Instead, all computing resources and allcache resources may retrieve data from, and store data to, any of thedata storage resources in the storage platform 210. Additionally, thedata processing platform 200 supports the addition of new computingresources and cache resources to the execution platform 116 withoutrequiring any changes to the storage platform 210. Similarly, the dataprocessing platform 200 supports the addition of data storage resourcesto the storage platform 210 without requiring any changes to nodes inthe execution platform 116.

FIG. 3 is a block diagram depicting an embodiment of the compute servicemanager 102. As shown in FIG. 3, the compute service manager 102includes an access manager 302 and a key manager 304 coupled to a datastorage device 306. Access manager 302 handles authentication andauthorization tasks for the systems described herein. Key manager 304manages storage and authentication of keys used during authenticationand authorization tasks. For example, access manager 302 and key manager304 manage the keys used to access data stored in remote storage devices(e.g., data storage devices in storage platform 210). As used herein,the remote storage devices may also be referred to as “persistentstorage devices” or “shared storage devices.” A request processingservice 308 manages received data storage requests and data retrievalrequests (e.g., jobs to be performed on database data). For example, therequest processing service 308 may determine the data necessary toprocess the received data storage request or data retrieval request. Thenecessary data may be stored in a cache within the execution platform116 (as discussed in greater detail below) or in a data storage devicein storage platform 210. A management console service 310 supportsaccess to various systems and processes by administrators and othersystem managers. Additionally, the management console service 310 mayreceive a request to execute a job and monitor the workload on thesystem.

The compute service manager 102 also includes a job compiler 312, a joboptimizer 314 and a job executor 310. The job compiler 312 parses a jobinto multiple discrete tasks and generates the execution code for eachof the multiple discrete tasks. The job optimizer 314 determines thebest method to execute the multiple discrete tasks based on the datathat needs to be processed. The job optimizer 314 also handles variousdata pruning operations and other data optimization techniques toimprove the speed and efficiency of executing the job. The job executor316 executes the execution code for jobs received from the queue 204 ordetermined by the compute service manager 102.

A job scheduler and coordinator 318 sends received jobs to theappropriate services or systems for compilation, optimization, anddispatch to the execution platform 116. For example, jobs may beprioritized and processed in that prioritized order. In an embodiment,the job scheduler and coordinator 318 determines a priority for internaljobs that are scheduled by the compute service manager 102 with other“outside” jobs such as user queries that may be scheduled by othersystems in the database but may utilize the same processing resources inthe execution platform 116. In some embodiments, the job scheduler andcoordinator 318 identifies or assigns particular nodes in the executionplatform 116 to process particular tasks. A virtual warehouse manager320 manages the operation of multiple virtual warehouses implemented inthe execution platform 116. As discussed below, each virtual warehouseincludes multiple execution nodes that each include a cache and aprocessor.

Additionally, the compute service manager 102 includes a configurationand metadata manager 322, which manages the information related to thedata stored in the remote data storage devices and in the local caches(i.e., the caches in execution platform 116). As discussed in greaterdetail below, the configuration and metadata manager 322 uses themetadata to determine which data files need to be accessed to retrievedata for processing a particular task or job. A monitor and workloadanalyzer 324 oversees processes performed by the compute service manager102 and manages the distribution of tasks (e.g., workload) across thevirtual warehouses and execution nodes in the execution platform 116.The monitor and workload analyzer 324 also redistributes tasks, asneeded, based on changing workloads throughout the data processingplatform 200 and may further redistribute tasks based on a user (i.e.“external”) query workload that may also be processed by the executionplatform 116. The configuration and metadata manager 322 and the monitorand workload analyzer 324 are coupled to a data storage device 326. Datastorage devices 306 and 326 in FIG. 3 represent any data storage devicewithin data processing platform 200. For example, data storage devices306 and 326 may represent caches in execution platform 116, storagedevices in storage platform 210, or any other storage device.

The compute service manager 102 also includes a transaction managementand access control module 328, which manages the various tasks and otheractivities associated with the processing of data storage requests anddata access requests. For example, transaction management and accesscontrol module 328 provides consistent and synchronized access to databy multiple users or systems. Since multiple users/systems may accessthe same data simultaneously, changes to the data must be synchronizedto ensure that each user/system is working with the current version ofthe data. Transaction management and access control module 328 providescontrol of various data processing activities at a single, centralizedlocation in the compute service manager 102. In some embodiments, thetransaction management and access control module 328 interacts with thejob executor 316 to support the management of various tasks beingexecuted by the job executor 316.

FIG. 4 is a block diagram depicting an embodiment of an executionplatform 116. As shown in FIG. 4, execution platform 116 includesmultiple virtual warehouses, including virtual warehouse 1, virtualwarehouse 2, and virtual warehouse n. Each virtual warehouse includesmultiple execution nodes that each include a data cache and a processor.The virtual warehouses can execute multiple tasks in parallel by usingthe multiple execution nodes. As discussed herein, execution platform116 can add new virtual warehouses and drop existing virtual warehousesin real-time based on the current processing needs of the systems andusers. This flexibility allows the execution platform 116 to quicklydeploy large amounts of computing resources when needed without beingforced to continue paying for those computing resources when they are nolonger needed. All virtual warehouses can access data from any datastorage device (e.g., any storage device in storage platform 210).

Although each virtual warehouse shown in FIG. 4 includes three executionnodes, a particular virtual warehouse may include any number ofexecution nodes. Further, the number of execution nodes in a virtualwarehouse is dynamic, such that new execution nodes are created whenadditional demand is present, and existing execution nodes are deletedwhen they are no longer necessary.

Each virtual warehouse is capable of accessing any of the data storagedevices 310 a-310 n shown in FIG. 3. Thus, the virtual warehouses arenot necessarily assigned to a specific data storage device 212 a-212 nand, instead, can access data from any of the data storage devices 212a-212 n within the storage platform 210. Similarly, each of theexecution nodes shown in FIG. 4 can access data from any of the datastorage devices 212 a-212 n. In some embodiments, a particular virtualwarehouse or a particular execution node may be temporarily assigned toa specific data storage device, but the virtual warehouse or executionnode may later access data from any other data storage device.

In the example of FIG. 4, virtual warehouse 1 includes three executionnodes 402 a, 402 b, and 402 n. Execution node 402 a includes a cache 404a and a processor 406 a. Execution node 402 b includes a cache 404 b anda processor 406 b. Execution node 402 n includes a cache 404 n and aprocessor 406 n. Each execution node 402 a, 402 b, and 402 n isassociated with processing one or more data storage and/or dataretrieval tasks. For example, a virtual warehouse may handle datastorage and data retrieval tasks associated with an internal service,such as a clustering service, a materialized view refresh service, afile compaction service, a storage procedure service, or a file upgradeservice. In other implementations, a particular virtual warehouse mayhandle data storage and data retrieval tasks associated with aparticular data storage system or a particular category of data.

Similar to virtual warehouse 1 discussed above, virtual warehouse 2includes three execution nodes 412 a, 412 b, and 412 n. Execution node412 a includes a cache 414 a and a processor 416 a. Execution node 412 bincludes a cache 414 b and a processor 416 b. Execution node 412 nincludes a cache 414 n and a processor 416 n. Additionally, virtualwarehouse 3 includes three execution nodes 422 a, 422 b, and 422 n.Execution node 422 a includes a cache 424 a and a processor 426 a.Execution node 422 b includes a cache 424 b and a processor 426 b.Execution node 422 n includes a cache 424 n and a processor 426 n.

In some embodiments, the execution nodes shown in FIG. 4 are statelesswith respect to the data the execution nodes are caching. For example,these execution nodes do not store or otherwise maintain stateinformation about the execution node, or the data being cached by aparticular execution node. Thus, in the event of an execution nodefailure, the failed node can be transparently replaced by another node.Since there is no state information associated with the failed executionnode, the new (replacement) execution node can easily replace the failednode without concern for recreating a particular state.

Although the execution nodes shown in FIG. 4 each include one data cacheand one processor, alternate embodiments may include execution nodescontaining any number of processors and any number of caches.Additionally, the caches may vary in size among the different executionnodes. The caches shown in FIG. 4 store, in the local execution node,data that was retrieved from one or more data storage devices in storageplatform 210. Thus, the caches reduce or eliminate the bottleneckproblems occurring in platforms that consistently retrieve data fromremote storage systems. Instead of repeatedly accessing data from theremote storage devices, the systems and methods described herein accessdata from the caches in the execution nodes which is significantlyfaster and avoids the bottleneck problem discussed above. In someembodiments, the caches are implemented using high-speed memory devicesthat provide fast access to the cached data. Each cache can store datafrom any of the storage devices in the storage platform 210.

Further, the cache resources and computing resources may vary betweendifferent execution nodes. For example, one execution node may containsignificant computing resources and minimal cache resources, making theexecution node useful for tasks that require significant computingresources. Another execution node may contain significant cacheresources and minimal computing resources, making this execution nodeuseful for tasks that require caching of large amounts of data. Yetanother execution node may contain cache resources providing fasterinput-output operations, useful for tasks that require fast scanning oflarge amounts of data. In some embodiments, the cache resources andcomputing resources associated with a particular execution node aredetermined when the execution node is created, based on the expectedtasks to be performed by the execution node.

Additionally, the cache resources and computing resources associatedwith a particular execution node may change over time based on changingtasks performed by the execution node. For example, an execution nodemay be assigned more processing resources if the tasks performed by theexecution node become more processor-intensive. Similarly, an executionnode may be assigned more cache resources if the tasks performed by theexecution node require a larger cache capacity.

Although virtual warehouses 1, 2, and n are associated with the sameexecution platform 116, the virtual warehouses may be implemented usingmultiple computing systems at multiple geographic locations. Forexample, virtual warehouse 1 can be implemented by a computing system ata first geographic location, while virtual warehouses 2 and n areimplemented by another computing system at a second geographic location.In some embodiments, these different computing systems are cloud-basedcomputing systems maintained by one or more different entities.

Additionally, each virtual warehouse is shown in FIG. 4 as havingmultiple execution nodes. The multiple execution nodes associated witheach virtual warehouse may be implemented using multiple computingsystems at multiple geographic locations. For example, an instance ofvirtual warehouse 1 implements execution nodes 402 a and 402 b on onecomputing platform at a geographic location and implements executionnode 402 n at a different computing platform at another geographiclocation. Selecting particular computing systems to implement anexecution node may depend on various factors, such as the level ofresources needed for a particular execution node (e.g., processingresource requirements and cache requirements), the resources availableat particular computing systems, communication capabilities of networkswithin a geographic location or between geographic locations, and whichcomputing systems are already implementing other execution nodes in thevirtual warehouse.

Execution platform 116 is also fault tolerant. For example, if onevirtual warehouse fails, that virtual warehouse is quickly replaced witha different virtual warehouse at a different geographic location.

A particular execution platform 116 may include any number of virtualwarehouses. Additionally, the number of virtual warehouses in aparticular execution platform is dynamic, such that new virtualwarehouses are created when additional processing and/or cachingresources are needed. Similarly, existing virtual warehouses may bedeleted when the resources associated with the virtual warehouse are nolonger necessary.

In some embodiments, the virtual warehouses may operate on the same datain storage platform 210, but each virtual warehouse has its ownexecution nodes with independent processing and caching resources. Thisconfiguration allows requests on different virtual warehouses to beprocessed independently and with no interference between the requests.This independent processing, combined with the ability to dynamicallyadd and remove virtual warehouses, supports the addition of newprocessing capacity for new users without impacting the performanceobserved by the existing users.

FIG. 5 is a block diagram depicting an example operating environment 500with the queue 204 in communication with multiple virtual warehousesunder a virtual warehouse manager 502. In environment 500, the queue 204has access to multiple database shared storage devices 506 a, 506 b, 506c, 506 d, 506 e and 506 n through multiple virtual warehouses 504 a, 504b, and 504 n. Although not shown in FIG. 5, the queue 204 may accessvirtual warehouses 504 a, 504 b, and 504 n through the compute servicemanager 102 (see FIG. 1). In particular embodiments, databases 506 a-506n are contained in the storage platform 210 and are accessible by anyvirtual warehouse implemented in the execution platform 116. In someembodiments, the queue 204 may access one of the virtual warehouses 504a-504 n using a data communication network such as the Internet. In someimplementations, a client account may specify that the queue 204(configured for storing internal jobs to be completed) should interactwith a particular virtual warehouse 504 a-504 n at a particular time.

In an embodiment (as illustrated), each virtual warehouse 504 a-504 ncan communicate with all databases 506 a-506 n. In some embodiments,each virtual warehouse 504 a-504 n is configured to communicate with asubset of all databases 506 a-506 n. In such an arrangement, anindividual client account associated with a set of data may send alldata retrieval and data storage requests through a single virtualwarehouse and/or to a certain subset of the databases 506 a-506 n.Further, where a certain virtual warehouse 504 a-504 n is configured tocommunicate with a specific subset of databases 506 a-506 n, theconfiguration is dynamic. For example, virtual warehouse 504 a may beconfigured to communicate with a first subset of databases 506 a-506 nand may later be reconfigured to communicate with a second subset ofdatabases 506 a-506 n.

In an embodiment, the queue 204 sends data retrieval, data storage, anddata processing requests to the virtual warehouse manager 502, whichroutes the requests to an appropriate virtual warehouse 504 a-504 n. Insome implementations, the virtual warehouse manager 502 provides adynamic assignment of jobs to the virtual warehouses 504 a-504 n.

In some embodiments, fault tolerance systems create a new virtualwarehouse in response to a failure of a virtual warehouse. The newvirtual warehouse may be in the same virtual warehouse group or may becreated in a different virtual warehouse group at a different geographiclocation.

The systems and methods described herein allow data to be stored andaccessed as a service that is separate from computing (or processing)resources. Even if no computing resources have been allocated from theexecution platform 116, data is available to a virtual warehouse withoutrequiring reloading of the data from a remote data source. Thus, data isavailable independently of the allocation of computing resourcesassociated with the data. The described systems and methods are usefulwith any type of data. In particular embodiments, data is stored in astructured, optimized format. The decoupling of the data storage/accessservice from the computing services also simplifies the sharing of dataamong different users and groups. As discussed herein, each virtualwarehouse can access any data to which it has access permissions, evenat the same time as other virtual warehouses are accessing the samedata. This architecture supports running queries without any actual datastored in the local cache. The systems and methods described herein arecapable of transparent dynamic data movement, which moves data from aremote storage device to a local cache, as needed, in a manner that istransparent to the user of the system. Further, this architecturesupports data sharing without prior data movement since any virtualwarehouse can access any data due to the decoupling of the data storageservice from the computing service.

FIG. 6 is a flow diagram depicting an example lifecycle 600 of a taskstate. The lifecycle 600 includes a series of task states beginning withthe states of generate task 602 and schedule task 604. The task statemay transition to schedule retrying 606 if the task needs to berescheduled after it is initially scheduled. During execution, the taskstate is referred to as execute task 608 and, upon execution, the taskstate may by any of: completed success 610, completed cancelled 612, orcompleted failed 614. In response to the task state being completedfailed 614 or completed cancelled 612, the task may be rescheduled andtransitioned back to a state of schedule retrying 606.

The task state is utilized to track where a given task is in itslifecycle 600. Transitions from one task state to a different task stateare illustrated by arrows in FIG. 6. The transitions are atomic andtriggered by the compute service manager 102. The task state may bestored as metadata and may be well-defined at all times.

In an implementation, there may be a need to perform task-specificactions outside the context of a SQL statement that is generated for ascheduled task. In an embodiment, a state transition callback isimplemented. The state transition callback may be defined by animplementer within a concrete task class and may be called by thecompute service manager 102 when a task is transitioned from one stateto another. A state transition is atomic but a call for a statetransition callback is not. As such, in the event of a fault, it may benecessary to call the state transition callback for a task for which thecallback has already been called and either partially or fullycompleted. Implementations of state transition callbacks may begenerated such that if a callback is run twice for a given task, thesystem is left in a consistent state. The state transition callback maybe run without a security context such that it cannot resolve an object.

FIG. 7 is a schematic block diagram of a process flow 700 for schedulingand executing a task on a database. The process flow 700 may beimplemented in a cloud-based computing environment that may include amultiple tenant database platform having storage and execution resourcesthat are separate from one another such that each is effectivelyinfinitely scalable. The process flow 700 may be implemented forexecuting jobs on the database system, where a job includes one or moretasks. The process flow 700 may be implemented by a compute servicemanager 102 in conjunction with an execution platform 116. The computeservice manager 102 may include the task dispatcher 706, the taskmanager 714, the query execution manager 716, the execution clustermanager 724, a module configured to create successor tasks 728, andother suitable components as described herein.

Units of compute service work may be divided into tasks. As describedherein, a task or other unit of compute service work may include an“internal” job to be completed by the compute service manager 102 suchthat the job is not directly requested by a client or client-facing.Such jobs may include, for example, clustering a table, refreshing amaterialized view, updating metadata, updating a secondary deployment toreflect updates made to a primary deployment, updating a change trackingsummary, and so forth. The units of compute service work may be dividedsuch that each task (see e.g. task 0, task 1, task 2, task 3 . . . taskn) includes enough information to run a Structured Query Language (SQL)query or perform some other piece of work. Tasks can be generated as adirect result of a manual action such as scheduling a SQL query to runat a fixed rate or may be programmatically generated such asautomatically reclustering a table in response to an update to thetable. The compute service infrastructure may serve as an elasticallyscalable means of managing and executing tasks.

The life cycle of a task begins when the task is generated. The task maybe generated based on the information that is needed to execute thetask. For some implementations, such as automatic clustering, automaticmaterialized view refresh, and so forth, this information includes SQLtext that is used to generate a query and the context for the query. Thecontext for the query may include a client account identification withinthe multiple tenant database system, a role, and/or a user to be used torun that query. The task is scheduled in a task queue 704 for execution.In an embodiment, the task queue 704 is stored in persistent storage.

An individual task in the task queue 704 remains queued until it isdequeued by the task dispatcher 706. As illustrated in FIG. 7, there mayexist multiple task dispatchers 706 and the system may effectivelyinclude an infinitely scalable number of task dispatchers 706 to serviceall database data across a multiple tenant database platform. The one ormore task dispatchers 706 may be scaled elastically to match throughputinto the task queue 704. Once a task has been dequeued, the taskdispatcher 706 receives the task at 708. The task dispatcher 706deserializes and resolves the execution context for the task at 710. Thetask dispatcher 706 may deserialize the information stored within thetask such as the SQL text and/or run context. The task dispatcher 706generates a SQL job for the task at 712. The task and SQL job record aresent to the query execution manager 716.

The query execution manager 716 receives the task and the SQL job at718. The query execution manager 716 parses, compiles, and optimizes theSQL job at 720 to generate a plan for executing the SQL job. The queryexecution manager 716 generates a query plan at 722 for the SQL job. Thequery plan may include a plurality of discrete subtasks that must becompleted to execute the task. The query plan is provided to theexecution cluster manager 724. The execution cluster manager 724 decideswhich execution node (may be referred to as a “cluster”) to execute thequery on, and the query is provided to that chosen execution node of theexecution platform 116. The chosen execution node executes the query.The query execution manager 716 may periodically communicate with thechosen execution node to perform any set of actions necessary for thequery and may specifically communicate with the chosen execution node tolearn when chosen execution node has finished executing the query.

The execution platform 116 is elastically scalable both in terms of asingle execution node to address query latency and the number ofexecution nodes to address query throughput. If the resources requiredfor a given query are large, the single execution node size might bescaled up to finish a query faster. Likewise, if the number of queriesrequiring execution increases, the number of execution nodes can beincreased to allow for more queries to be executed at one time.

When the query has finished executing, the query execution manager 716signals to the task manager 714 that the task has been completed. Thetask manager 714 uses the information stored in the original task (seetask 0, task 1, task 2, task 3 . . . task n) to programmaticallygenerate a series of successor tasks 732 such as task 0′, task 1′, task2′, task 3′ . . . task n′. The successor tasks 732 may be a continuationof a parent task or may do something entirely different. Once generated,the successor tasks 732 are scheduled in the tasks queue 704 and theoriginal task is transition to a completed task 730 state. Thescheduling of the successor task 732 in the task queue 704 and thetransitioning of the original task to the completed task 730 state mayhappen atomically such that if completing the task must be retried,duplicate successor tasks will not be generated. Once completed, a taskis persistently stored in the completed task 730 storage for a setduration to allow task history to be audited for a limited duration. Invarious implementations, the completed task 730 storage may exist forany suitable duration of time and may or may not be cleared or refreshedbased on a time duration.

In an embodiment, the task dispatcher 706, the task manager 714, and thequery execution manager 716 are all located on the same physical serverand may be scaled elastically to meet task throughput by scaling thenumber of instances. In an embodiment, the scaling of task dispatcher706 instances, task manager 714 instances, and query execution manager716 instances is 1:1 with the number of compute service manager 102instances.

FIG. 8 is a flow diagram depicting an embodiment of a method 800 formanaging data storage and retrieval operations. The method 800 may beperformed by any suitable computing device or system, including forexample a compute service manager 102 or data processing platform 200 asdisclosed herein.

The method 800 begins and a computing device determines at 802 a job tobe processed on database data, wherein the job is determined in responseto a trigger event. The computing device determines at 804 a pluralityof discrete tasks that must be processed to complete the job. Thecomputing device divides at 806 processing of the plurality of discretetasks. The computing device assigns at 808 different tasks of theplurality of discrete tasks to different nodes within a plurality ofnodes of an execution platform. The computing device executes at 810, bythe execution platform, the plurality of discrete tasks, wherein eachnode of the plurality of nodes is independent of and remote from aplurality of shared storage devices.

FIG. 9 is a flow diagram depicting an embodiment of a method 900 formanaging data storage and retrieval operations for an internal databasejob. The method 900 may be performed by any suitable computing device orsystem, including for example a compute service manager 102 or dataprocessing platform 200 as disclosed herein.

The method 900 begins and a computing device determines at 902 a job tobe processed on data of a database, wherein the job is determined inresponse to a trigger event and the job is configured to improve dataorganization of the database or improve query performance for thedatabase. In an embodiment, the job is determined internally to thedatabase and does not include a query received from a user or clientaccount. The job may include, for example, clustering or reclustering adatabase table, refreshing a materialized view that has become stalewith respect to its source table, compacting one or more database rowsor tables, executed a storage procedure, upgrading a file ormicro-partition of database table, and so forth.

The method 900 continues and the computing device identifies at 904 oneor more micro-partitions within the database that must be processed toexecute the job. In an embodiment, the database includes a plurality oftables that each include one or more immutable storage devices referredto as a micro-partition, wherein the micro-partition cannot be updatedin-place but must instead be recreated in response to, for example, aninsert, delete, or update DML (Data Manipulation Language) command. Insuch an embodiment, the computing device may identify, for example, thatone or more materialized view micro-partitions must be regenerated tosync the materialized view with its source table. Further in such anembodiment, the computing device may identify, for example, one or moremicro-partitions that may be spread across one or more database tablesthat must be reclustered to complete a clustering job.

The method 900 continues and the computing device divides at 906processing of the one or more micro-partitions into multiple discretetasks. Examples of such multiple discrete tasks include inserting a rowinto a table, deleting a row from a table, updating a row in a table,updating metadata about a table, updating a change tracking stream abouta table, performing a clustering operation, comparing a materializedview with its source table, upgrading a micro-partition, and so forth.

The method 900 continues and the computing device communicates at 908with a resource manager that is configured to schedule user queries onthe database. The computing device communicates at 908 with the resourcemanager specifically to determine a current or future workload for oneor more execution nodes of a plurality of execution nodes in anexecution platform. In such an embodiment, the computing device may be acompute service manager and the resource manager may be independent ofand remote from the compute service manager. Thus, the resource managermay be configured to schedule and manage the execution of user queriesand the compute service manager may be configured to schedule and managethe execution of internal jobs that improve performance or organizationof the database. The method 900 continues and the computing deviceassigns at 910 different tasks of the multiple discrete tasks todifferent nodes within a plurality of nodes of the execution platform.The computing device assigns at 910 based at least in part on thecurrent or future workload for the one or more execution nodes of theplurality of execution nodes in the execution platform, as received fromthe resource manager. In an embodiment, user queries that are scheduledby the resource manager may take priority over internal jobs. The method900 continues the execution platform executes at 912 the multiplediscrete tasks, wherein each node of the plurality of nodes of theexecution platform is independent of and remote from a plurality ofshared storage devices collectively storing the data for the database.In an embodiment, each node of the execution platform and each sharedstorage device is independent and remote such that the executionplatform may be scaled up or down independent of data storage.

In some situations, users (or system administrators) may desireincreased performance (e.g., increased query response time). In thesesituations, additional virtual warehouses may be added to support thisincreased performance. In other implementations, the compute servicemanager predicts upcoming resource needs based on scheduled (but not yetexecuted) jobs or queries. In an embodiment, the compute service managerreceives a log of scheduled queries from a resource manager, wherein theresource manager is configured to schedule and manage the execution ofuser queries received from one or more client accounts. In such anembodiment, client queries may take priority over internal jobs, and thecompute service manager may schedule internal jobs to provide sufficientprocessing capacity for client queries that are scheduled by theresource manager. If the scheduled queries will significantly degradethe system's performance, the compute service manager and/or theresource manager can add more resources prior to executing those queriesor jobs, thereby maintaining overall system performance. After thosequeries or jobs are executed, the added resources can be deactivated bythe compute service manager and/or the resource manager.

In some embodiments, the compute service manager predicts a timerequired to execute a particular job (or group of jobs). Based oncurrent job/query processing performance (e.g., processing delays,system utilization, etc.), the compute service manager determineswhether additional resources are needed for that particular job or groupof jobs. For example, if the current processing delay exceeds athreshold value, the compute service manager may create one or more newexecution nodes to provide additional resources for processing theparticular job or group of jobs. After processing of the job or group ofjobs is complete, the compute service manager may deactivate the newexecution node(s) if they are no longer needed for processing otherjobs.

In some embodiments, a particular user may require certain performancelevels when processing the user's queries. For example, the user mayrequire a query response within a particular time period, such as 5seconds. In these embodiments, the compute service manager may allocateadditional resources prior to executing an internal job to ensure theuser's performance levels are achieved.

As described herein, data processing platform 200 supports the dynamicactivation and deactivation of various resources, such as data storagecapacity, processing resources, cache resources, and the like. Thesingle data processing platform 200 can be dynamically changed on-demandbased on the current data storage and processing requirements of thepending and anticipated data processing requests. As the data storageand processing requirements change, data processing platform 200automatically adjusts to maintain a substantially uniform level of dataprocessing performance.

Additionally, the described data processing platform 200 permits changesto the data storage capacity and the processing resources independently.For example, the data storage capacity can be modified without makingany changes to the existing processing resources. Similarly, theprocessing resources can be modified without making any changes to theexisting data storage capacity.

In some implementations, the same micro-partition is cached by multipleexecution nodes at the same time. This multiple caching ofmicro-partitions helps with load balancing (e.g., balancing dataprocessing tasks) across multiple execution nodes. Additionally, cachinga micro-partition in multiple execution nodes helps avoid potentialbottlenecks when significant amounts of data are trying to pass throughthe same communication link. This implementation also supports theparallel processing of the same data by different execution nodes.

The systems and methods described herein take advantage of the benefitsof both shared-disk systems and the shared-nothing architecture. Thedescribed platform for storing and retrieving data is scalable like theshared-nothing architecture once data is cached locally. It also has allthe benefits of a shared-disk architecture where processing nodes can beadded and removed without any constraints (e.g., for 0 to N) and withoutrequiring any explicit reshuffling of data.

FIG. 10 is a block diagram depicting an example computing device 1000.In some embodiments, computing device 1000 is used to implement one ormore of the systems and components discussed herein. For example,computing device 1000 may allow a user or administrator to accesscompute service manager 102. Further, computing device 1000 may interactwith any of the systems and components described herein. Accordingly,computing device 1000 may be used to perform various procedures andtasks, such as those discussed herein. Computing device 1000 canfunction as a server, a client or any other computing entity. Computingdevice 1000 can be any of a wide variety of computing devices, such as adesktop computer, a notebook computer, a server computer, a handheldcomputer, a tablet, and the like.

Computing device 1000 includes one or more processor(s) 1002, one ormore memory device(s) 1004, one or more interface(s) 1006, one or moremass storage device(s) 1008, and one or more Input/Output (I/O)device(s) 1010, all of which are coupled to a bus 1012. Processor(s)1002 include one or more processors or controllers that executeinstructions stored in memory device(s) 1004 and/or mass storagedevice(s) 1008. Processor(s) 1002 may also include various types ofcomputer-readable media, such as cache memory.

Memory device(s) 1004 include various computer-readable media, such asvolatile memory (e.g., random access memory (RAM)) and/or nonvolatilememory (e.g., read-only memory (ROM)). Memory device(s) 1004 may alsoinclude rewritable ROM, such as Flash memory.

Mass storage device(s) 1008 include various computer readable media,such as magnetic tapes, magnetic disks, optical disks, solid statememory (e.g., Flash memory), and so forth. Various drives may also beincluded in mass storage device(s) 1008 to enable reading from and/orwriting to the various computer readable media. Mass storage device(s)1008 include removable media and/or non-removable media.

I/O device(s) 1010 include various devices that allow data and/or otherinformation to be input to or retrieved from computing device 1000.Example I/O device(s) 1010 include cursor control devices, keyboards,keypads, microphones, monitors or other display devices, speakers,printers, network interface cards, modems, lenses, CCDs or other imagecapture devices, and the like.

Interface(s) 1006 include various interfaces that allow computing device1000 to interact with other systems, devices, or computing environments.Example interface(s) 1006 include any number of different networkinterfaces, such as interfaces to local area networks (LANs), wide areanetworks (WANs), wireless networks, and the Internet.

Bus 1012 allows processor(s) 1002, memory device(s) 1004, interface(s)1006, mass storage device(s) 1008, and I/O device(s) 1010 to communicatewith one another, as well as other devices or components coupled to bus1012. Bus 1012 represents one or more of several types of busstructures, such as a system bus, PCI bus, IEEE 1394 bus, USB bus, andso forth.

For purposes of illustration, programs and other executable programcomponents are shown herein as discrete blocks, although it isunderstood that such programs and components may reside at various timesin different storage components of computing device 1000 and areexecuted by processor(s) 1002. Alternatively, the systems and proceduresdescribed herein can be implemented in hardware, or a combination ofhardware, software, and/or firmware. For example, one or moreapplication specific integrated circuits (ASICs) can be programmed tocarry out one or more of the systems and procedures described herein.

EXAMPLES

The following examples pertain to further embodiments:

Example 1 is a system for scheduling and managing a job to be performedon database data. The system includes a plurality of shared storagedevices collectively storing database data. The system includes acompute service manager comprising a software program stored in memoryand executed by one or more processors, the compute service managerconfigured to: determine a job to be processed on the database data,wherein the job is determined in response to a trigger event; determinea plurality of discrete tasks that must be processed to complete thejob; divide processing of the plurality of discrete tasks; and assigndifferent tasks of the plurality of discrete tasks to different nodeswithin a plurality of nodes of an execution platform. The system is suchthat the execution platform configured to execute the plurality ofdiscrete tasks, wherein each node of the plurality of nodes isindependent of and remote from the plurality of shared storage devices.

Example 2 is a system as in Example 1, wherein the execution platformcomprises: at least one processor executing one or more tasks of theplurality of discrete tasks assigned thereto by the compute servicemanager; and at least one local cache caching at least a portion of thedatabase data.

Example 3 is a system as in any of Examples 1-2, wherein the databasedata is stored in a plurality of tables, wherein each of the pluralityof tables comprises one or more immutable storage devices such that anupdate to at least one of the one or more immutable storage devicescomprises recreating the at least one immutable storage device.

Example 4 is a system as in any of Examples 1-3, further comprising aresource manager configured to schedule and manage execution of queriesreceived from a client account by assigning tasks for completing thequeries to one or more nodes of the plurality of nodes of the executionplatform, wherein the compute service manager is independent of theresource manager, and wherein the job determined by the compute servicemanager comprises an internal job for improving database performance ororganization of the database data and does not include a query receivedfrom a client account.

Example 5 is a system as in any of Examples 1-4, wherein the jobcomprises one or more of: reclustering database data; refreshing amaterialized view; or compacting database data.

Example 6 is a system as in any of Examples 1-5, wherein the executionplatform asynchronously executes the plurality of discrete tasks, andwherein two or more of the plurality of discrete tasks are executed inparallel by two or more of the plurality of nodes of the executionplatform.

Example 7 is a system as in any of Examples 1-6, wherein the triggerevent comprises one or more of: ingesting new data into a databasetable; deleting one or more rows of a database table; updating one ormore rows of a database table; a clustering service determining that oneor more database tables need to be reclustered; or a refreshmaterialized view service determining that a materialized view is notsynced with its source table.

Example 8 is a system as in any of Examples 1-7, further comprising aqueue configured to store one or more jobs to be performed on thedatabase data, wherein the compute service manager is configured todetermine the job to be processed by querying the queue to determinewhether any unprocessed jobs are available at the queue.

Example 9 is a system as in any of Examples 1-8, wherein the computeservice manager is configured to assign different tasks of the pluralityof discrete tasks based on one or more of: a schedule defined by aclient account; a current workload for the execution platform, whereinthe workload comprises one or more of: jobs assigned by the computeservice manager or queries received from a client account; or a ruledefined by a client account.

Example 10 is a system as in any of Examples 1-9, wherein the computeservice manager is further configured to: determine whether to assign asuccessor to a task; and in response to determining to assign thesuccessor to the task, generating parameters for the successor andplacing the successor in a queue.

Example 11 is a method for scheduling and managing jobs to be executedon a database. The method includes determining a job to be processed ondatabase data, wherein the job is determined in response to a triggerevent. The method includes determining a plurality of discrete tasksthat must be processed to complete the job. The method includes dividingprocessing of the plurality of discrete tasks and assigning differenttasks of the plurality of discrete tasks to different nodes within aplurality of nodes of an execution platform. The method is such that theexecution platform is configured to execute the plurality of discretetasks, wherein each node of the plurality of nodes is independent of andremote from a plurality of shared storage devices collectively storingthe database data.

Example 12 is a method as in Example 11, wherein the execution platformcomprises: at least one processor executing one or more tasks of theplurality of discrete tasks assigned thereto by the compute servicemanager; and at least one local cache caching at least a portion of thedatabase data.

Example 13 is a method as in any of Examples 11-12, wherein the databasedata is stored in a plurality of tables, wherein each of the pluralityof tables comprises one or more immutable storage devices such that anupdate to at least one of the one or more immutable storage devicescomprises recreating the at least one immutable storage device.

Example 14 is a method as in any of Examples 11-14, further comprisingcommunicating with a resource manager configured to schedule and manageexecution of queries received from a client account and wherein theresource manager further assigns tasks to the plurality of nodes of theexecution platform, and wherein the job comprises an internal job forimproving database performance or organization of the database data anddoes not include a query received from a client account.

Example 15 is a method as in any of Examples 11-14, wherein the jobcomprises one or more of: reclustering database data; refreshing amaterialized view; or compacting database data.

Example 16 is a method as in any of Examples 11-15, wherein the triggerevent comprises one or more of: ingesting new data into a databasetable; deleting one or more rows of a database table; updating one ormore rows of a database table; a clustering service determining that oneor more database tables need to be reclustered; or a refreshmaterialized view service determining that a materialized view is notsynced with its source table.

Example 17 is a method as in any of Examples 11-16, wherein determiningthe job to be processed comprises querying a queue to determine whetherany unprocessed jobs are available at the queue, wherein each of theunprocessed jobs are generated based on a trigger event.

Example 18 is a method as in any of Examples 11-17, wherein assigningdifferent tasks of the plurality of discrete tasks comprises assigningbased on one or more of: a schedule defined by a client account; acurrent workload for the execution platform, wherein the workloadcomprises one or more of: jobs assigned by the compute service manageror queries received from a client account; or a rule defined by a clientaccount.

Example 19 is a method as in any of Examples 11-18, further comprising:determining whether to assign a successor to a task; and in response todetermining to assign the successor to the task, generate parameters forthe successor and place the successor in a queue.

Example 20 is a system for scheduling and managing jobs to be executedon a database. The system includes means for storing database data andmeans for determining a job to be processed on the database data,wherein the job is determined in response to a trigger event. The systemincludes means for determining a plurality of discrete tasks that mustbe processed to complete the job. The system includes means for dividingprocessing of the plurality of discrete tasks and means for assigningdifferent tasks of the plurality of discrete tasks to different nodeswithin a plurality of nodes of an execution platform. The system is suchthat the execution platform executing the plurality of discrete tasks,wherein each node of the plurality of nodes is independent of and remotefrom the means for storing database data.

Example 21 is a system as in Example 20, further comprising means forscheduling and managing execution of queries received from a clientaccount by assigning tasks for completing queries to one or more nodesof the execution platform, wherein the means for scheduling and managingexecution of queries is independent of the means for determining thejob, and wherein the job comprises an internal job for improvingdatabase performance or organization of the database data and does notinclude a query received from a client account.

Example 22 is a system as in any of Examples 20-21, wherein the jobcomprises one or more of: reclustering database data; refreshing amaterialized view; or compacting database data.

Example 23 is a system as in any of Examples 20-22, wherein the triggerevent comprises one or more of: ingesting new data into a databasetable; deleting one or more rows of a database table; updating one ormore rows of a database table; a clustering service determining that oneor more database tables need to be reclustered; or a refreshmaterialized view service determining that a materialized view is notsynced with its source table.

Example 24 is a system as in any of Examples 20-23, wherein the meansfor assigning different tasks of the plurality of discrete tasks isconfigured to assign different tasks based on one or more of: a scheduledefined by a client account; a current workload for the executionplatform, wherein the workload comprises one or more of: jobs assignedby the compute service manager or queries received from a clientaccount; or a rule defined by a client account.

Example 25 is a system for managing and executing database tasks. Thesystem includes means for determining a task to be executed in responseto a trigger event. The system includes means for determining a queryplan for executing the task, wherein the query plan comprises aplurality of discrete subtasks. The system includes means for assigningthe plurality of discrete subtasks to one or more nodes of a pluralityof nodes of an execution platform. The system includes means fordetermining whether execution of the task is complete. The systemincludes means for storing a record of a completed task in response todetermining the execution of the task is complete.

Example 26 is a system as in Example 25, further comprising: means forgenerating a successor task in response to determining the execution ofthe task is complete; and means for persisting the successor task to atask queue.

Example 27 is a system as in any of Examples 25-26, further comprisingmeans for retrieving the task from a task queue, wherein the task queueis stored across one or more of a plurality of shared storage devicesthat are separate from the execution platform.

Example 28 is a system as in any of Examples 25-27, further comprisingmeans for scaling a number of available execution nodes in the executionplatform up or down based on a number of tasks waiting in the taskqueue.

Example 29 is a system as in any of Examples 25-28, further comprisingmeans for deserializing the task to resolve a context for the task,wherein the context comprises one or more of an account identification,a role, or a user for executing the task.

Example 30 is a system as in any of Examples 25-29, wherein the meansfor determining the query plan for executing the task is configured togenerate a Structured Query Language (SQL) job, and wherein theplurality of discrete subtasks must be processed to complete the SQLjob.

Example 31 is a system as in any of Examples 25-30, further comprisingmeans for dividing the processing of the plurality of discrete subtasksbased on metadata, wherein the metadata pertains to database data thatis stored across a plurality of shared storage devices, and wherein themetadata is stored separately from the database data.

Example 32 is a system as in any of Examples 25-31, wherein the meansfor storing the record of the completed task is configured to store therecord in persistent storage and is further configured to purge therecord from the persistent storage after a threshold duration of time.

Example 33 is a system as in any of Examples 25-32, further comprisingmeans for executing each of the plurality of discrete subtasks such thattwo or more of the plurality of discrete subtasks are executed inparallel by two or more execution nodes of the execution platform.

Example 34 is a system as in any of Examples 25-33, wherein the triggerevent comprises one or more of: ingesting new data into a databasetable; deleting one or more rows of a database table; updating one ormore rows of a database table; a clustering service determining that oneor more database tables need to be reclustered; or a refreshmaterialized view service determining that a materialized view is notsynced with its source table.

Example 35 is a method for managing and executing database tasks. Themethod includes determining a task to be executed in response to atrigger event. The method includes determining a query plan forexecuting the task, wherein the query plan comprises a plurality ofdiscrete subtasks. The method includes assigning the plurality ofdiscrete subtasks to one or more nodes of a plurality of nodes of anexecution platform. The method includes determining whether execution ofthe task is complete. The method includes storing a record of acompleted task in response to determining the execution of the task iscomplete.

Example 36 is a method as in Example 35, further comprising: generatinga successor task in response to determining the execution of the task iscomplete; and persisting the successor task to a task queue.

Example 37 is a method as in any of Examples 35-36, further comprisingretrieving the task from a task queue, wherein the task queue is storedacross one or more of a plurality of shared storage devices that areseparate from the execution platform.

Example 38 is a method as in any of Examples 35-37, further comprisingscaling a number of available execution nodes in the execution platformup or down based on a number of tasks waiting in the task queue.

Example 39 is a method as in any of Examples 35-38, further comprisingdeserializing the task to resolve a context for the task, wherein thecontext comprises one or more of an account identification, a role, or auser for executing the task.

Example 40 is a method as in any of Examples 35-39, wherein determiningthe query plan for executing the task comprises generating a StructuredQuery Language (SQL) job, wherein the plurality of discrete subtasksmust be processed to complete the SQL job.

Example 41 is a method as in any of Examples 35-40, further comprisingdividing the processing of the plurality of discrete subtasks based onmetadata, wherein the metadata pertains to database data that is storedacross a plurality of shared storage devices, and wherein the metadatais stored separately from the database data.

Example 42 is a method as in any of Examples 35-41, wherein storing therecord of the completed task comprises storing the record in persistentstorage and further comprises purging the record from the persistentstorage after a threshold duration of time.

Example 43 is a method as in any of Examples 35-42, wherein the triggerevent comprises one or more of: ingesting new data into a databasetable; deleting one or more rows of a database table; updating one ormore rows of a database table; a clustering service determining that oneor more database tables need to be reclustered; or a refreshmaterialized view service determining that a materialized view is notsynced with its source table.

Example 44 is a system. The system includes a plurality of sharedstorage devices collectively storing database data. The system includesa compute service manager comprising a software program stored in memoryand executed by one or more processors, the compute service managerconfigured to: determine a task to be executed in response to a triggerevent; determine a query plan for executing the task, the query plancomprising a plurality of discrete subtasks; assign the plurality ofdiscrete subtasks to one or more nodes of a plurality of nodes of anexecution platform; determine whether execution of the task is complete;and in response to determining the execution of the task is complete,storing a record indicating the task was completed; the executionplatform configured to execute the plurality of discrete subtasks,wherein each node of the plurality of nodes of the execution platform isindependent of and remote from the plurality of shared storage devices.

Example 45 is a system as in Example 44, wherein the compute servicemanager is further configured to: retrieve the task from a task queue,wherein the task queue is stored across the plurality of shared storagedevices; generate a successor task in response to determining theexecution of the task is complete; and persist the successor task to thetask queue.

Example 46 is a system as in any of Examples 44-45, wherein the computeservice manager is further configured to scale a number of availableexecution nodes in the execution platform up or down based on a numberof tasks within in the task queue.

Example 47 is a system as in any of Examples 44-46, wherein the computeservice manager is further configured to: deserialize the task toresolve a context for the task, wherein the context comprises one ormore of an account identification, a role, or a user for executing thetask; and divide the processing of the plurality of discrete subtasksbased on metadata, wherein the metadata pertains to the database dataand is stored separately from the database data.

Example 48 is a system as in any of Examples 44-47, wherein the computeservice manager is configured to store the record of the completed taskin persistent storage and is further configured to purge the record fromthe persistent storage after a threshold duration of time.

Many of the functional units described in this specification may beimplemented as one or more components, which is a term used to moreparticularly emphasize their implementation independence. For example, acomponent may be implemented as a hardware circuit comprising customvery large-scale integration (VLSI) circuits or gate arrays,off-the-shelf semiconductors such as logic chips, transistors, or otherdiscrete components. A component may also be implemented in programmablehardware devices such as field programmable gate arrays, programmablearray logic, programmable logic devices, or the like.

Components may also be implemented in software for execution by varioustypes of processors. An identified component of executable code may, forinstance, comprise one or more physical or logical blocks of computerinstructions, which may, for instance, be organized as an object, aprocedure, or a function. Nevertheless, the executables of an identifiedcomponent need not be physically located together but may comprisedisparate instructions stored in different locations that, when joinedlogically together, comprise the component and achieve the statedpurpose for the component.

Indeed, a component of executable code may be a single instruction, ormany instructions, and may even be distributed over several differentcode segments, among different programs, and across several memorydevices. Similarly, operational data may be identified and illustratedherein within components and may be embodied in any suitable form andorganized within any suitable type of data structure. The operationaldata may be collected as a single data set or may be distributed overdifferent locations including over different storage devices, and mayexist, at least partially, merely as electronic signals on a system ornetwork. The components may be passive or active, including agentsoperable to perform desired functions.

Reference throughout this specification to “an example” means that afeature, structure, or characteristic described in connection with theexample is included in at least one embodiment of the presentdisclosure. Thus, appearances of the phrase “in an example” in variousplaces throughout this specification are not necessarily all referringto the same embodiment.

As used herein, a plurality of items, structural elements, compositionalelements, and/or materials may be presented in a common list forconvenience. However, these lists should be construed as though eachmember of the list is individually identified as a separate and uniquemember. Thus, no individual member of such list should be construed as ade facto equivalent of any other member of the same list solely based onits presentation in a common group without indications to the contrary.In addition, various embodiments and examples of the present disclosuremay be referred to herein along with alternatives for the variouscomponents thereof. It is understood that such embodiments, examples,and alternatives are not to be construed as de facto equivalents of oneanother but are to be considered as separate and autonomousrepresentations of the present disclosure.

Although the foregoing has been described in some detail for purposes ofclarity, it will be apparent that certain changes and modifications maybe made without departing from the principles thereof. It should benoted that there are many alternative ways of implementing both theprocesses and apparatuses described herein. Accordingly, the presentembodiments are to be considered illustrative and not restrictive.

Those having skill in the art will appreciate that many changes may bemade to the details of the above-described embodiments without departingfrom the underlying principles of the disclosure. The scope of thepresent disclosure should, therefore, be determined only by thefollowing claims.

What is claimed is:
 1. A system comprising: one or more computerprocessors; and one or more computer-readable mediums storinginstructions that, when executed by the one or more computer processors,cause the system to perform operations comprising: in response todetermining that a materialized view of a database table is stale,determining a database task to be executed in a database for providing arefreshed materialized view of the database table, the database taskcomprising a plurality of discrete subtasks; storing the database taskat an execution platform that is distinct from the system; assigning theplurality of discrete subtasks to one or more nodes of a plurality ofnodes of the execution platform; dividing processing of the plurality ofdiscrete subtasks based on metadata of database data; determiningwhether execution of the database task is complete based at least inpart on whether the plurality of discrete subtasks have been completedby the one or more nodes; and in response to determining that executionof the database task is complete, storing a record indicating that thedatabase task is completed.
 2. The system of claim 1, the operationsfurther comprising: generating a successor database task in response todetermining that execution of the database task is complete.
 3. Thesystem of claim 2, the operations further comprising: persisting thesuccessor database task to a task queue.
 4. The system of claim 1, theoperations further comprising: retrieving the database task from a taskqueue, wherein the task queue comprises the plurality of discretesubtasks.
 5. The system of claim 4, the operations further comprising:scaling a number of available execution nodes in the execution platformup or down based on a number of database tasks remaining in the taskqueue.
 6. The system of claim 1, the operations further comprising:deserializing the database task to resolve a context for the databasetask, wherein the context comprises one or more of an accountidentification, a role, or a user for executing the database task. 7.The system of claim 1, wherein determining the database task comprisesgenerating a Structured Query Language (SQL) job, and wherein theplurality of discrete subtasks are processed to complete the SQL job. 8.The system of claim 1, storing the record indicating that the databasetask is completed further comprises storing the record in persistentstorage and purging the record from the persistent storage after athreshold duration of time.
 9. The system of claim 1, the operationsfurther comprising: executing each of the plurality of discrete subtaskssuch that two or more of the plurality of discrete subtasks are executedin parallel by two or more execution nodes of the execution platform.10. A method comprising: in response to determining that a materializedview of a database table is stale, determining, by a computing system, adatabase task to be executed in a database for providing a refreshedmaterialized view of the database table, the database task comprising aplurality of discrete subtasks; storing the database task at anexecution platform that is distinct from the computing system; assigningthe plurality of discrete subtasks to one or more nodes of a pluralityof nodes of the execution platform; dividing processing of the pluralityof discrete subtasks based on metadata of database data; determiningwhether execution of the database task is complete based at least inpart on whether the plurality of discrete subtasks have been completedby the one or more nodes; and in response to determining that executionof the database task is complete, storing a record indicating that thedatabase task is completed.
 11. The method of claim 10, furthercomprising: generating a successor database task in response todetermining that execution of the database task is complete.
 12. Themethod of claim 11, further comprising: persisting the successordatabase task to a task queue.
 13. The method of claim 10, furthercomprising: retrieving the database task from a task queue, wherein thetask queue comprises the plurality of discrete subtasks.
 14. The methodof claim 13, further comprising: scaling a number of available executionnodes in the execution platform up or down based on a number of databasetasks remaining in the task queue.
 15. The method of claim 14, furthercomprising: deserializing the database task to resolve a context for thedatabase task, wherein the context comprises one or more of an accountidentification, a role, or a user for executing the database task. 16.The method of claim 10, wherein determining the database task comprisesgenerating a Structured Query Language (SQL) job, and wherein theplurality of discrete subtasks are processed to complete the SQL job.17. The method of claim 10, storing the record indicating that thedatabase task is completed further comprises storing the record inpersistent storage and purging the record from the persistent storageafter a threshold duration of time.
 18. The method of claim 10, furthercomprising: executing each of the plurality of discrete subtasks suchthat two or more of the plurality of discrete subtasks are executed inparallel by two or more execution nodes of the execution platform.
 19. Anon-transitory computer-readable medium storing instructions that, whenexecuted by one or more computer processors of one or more computingdevices, cause the one or more computing devices to perform operationscomprising: in response to determining that a materialized view of adatabase table is stale, determining a database task to be executed in adatabase for providing a refreshed materialized view of the databasetable, the database task comprising a plurality of discrete subtasks;storing the database task at an execution platform that is distinct fromthe one or more computing devices; assigning the plurality of discretesubtasks to one or more nodes of a plurality of nodes of the executionplatform; dividing processing of the plurality of discrete subtasksbased on metadata of database data; determining whether execution of thedatabase task is complete based at least in part on whether theplurality of discrete subtasks have been completed by the one or morenodes; and in response to determining that execution of the databasetask is complete, storing a record indicating that the database task iscompleted.
 20. The non-transitory computer-readable medium of claim 19,the operations further comprising: generating a successor database taskin response to determining that execution of the database task iscomplete.